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ABSTRACT 

An analytical technique from the field of market 
research called conjoint analysis was applied to a psychological 
measurement of student testing design preferences. Past concerns with 
testing design are reviewed, and a newer approach to testing is 
identified— the modified confidence weighted-admissible probability 
measurement (MCW-AFM) test scoring method developed by J. Bruno 
(1985) . The conjoint analysis approach was applied to measure student 
preferences for testing design. T,ie conjoint design was developed 
With four factors: objectivity (subjective and objective), 
restrictions (restricted and unrestricted) , partial credit (allowed 
and not allowed) , and feedback (given and not given) . Usable survey 
responses were received from 516 individuals from different levels of 
education and different colleges. To demonstrate the flexibility of 
conjoint analysis, the study design was mocified to add a fifth 
factor of test-responding simplicity. The .illustration indicates that 
conjoint analysis can be used for larger sample sizes and that it 
permits analysis by demographic factors, a feature that is applicable 
to education. Six tables present study data. A 12-item list of 
references is included. (SLD) 
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A PSYCHOLOGICAL MEASUREMENT OF STUDENT 
TESTING DESIGN PREFERENCES 

ABSTRACT 

This study describes an analytical technique from the field of market re- 
search called conjoint analysis applied to a psychological measurement of student 
testing design preferences. The study reviews past concerns with testing design 
and identifies a newer approach to testing: the modified confidence weighted- 
admissible probability measurement (MCW-APM) test scoring method developed 
by Bruno (1985). The conjoint analysis approach is applied to measure student 
preferences for testing design. 
Introduction 

Conjoint analysis is a technique which has been widely used in marketing 
for several different commercial applications: new product design, consumer 
behavior studies, pricing decisions, sales training, and others (Cattin, 1982; 
Winnick, 1989). Unfortunately, the conjoint analysis technique has received 
very little attention in the education literature. There have been applications of 
the conjoint procedure to curriculum development (Sparling, 1977), student's 
university choice processes (f'ooley, 1981), library services preferences (Rams- 
ing, 1982), faculty preferences for collective bargaining (Shukla, 1990), and 
course design (Zufryden, 1983). 
Application of Coi\ioint Methods to Testing Design 

As identified by authors over a number of years, there are concerns about 
the accuracy of student knowledge as assessed by convent) onaUy designed and 
scored standardized student tests (Coombs, 1956;Bruno, 1985). It has been 
demonstrated through empirical studies that testing may not represent true knowl- 
edge due to biases introduced from guessing, wording of options provided for re- 



sponses, constrained choices for response, and right or wrong scoring without the 
possibility of partial credit (Bruno, 1989). Given the inaccuracies possible with 
such designed tests, researchers have examined the costs and benefits of testing 
(Catterall, 1982;Catterall, 1988). Catterall identified and explored the direct and 
indirect monetary costs as well as the opportunity and psychological costs of 
testing in an inner city and suburban elementary school (Catterall, 1982). As a 
result of the deficiencies found with conventional forced choice or binary type 
(right-wrong) test scoring methods, alternate designs have been developed and 
applied to actual student populations. One newer approach is the modified confi- 
dence weighted-admissible probability measurement (MCW-APM) test scoring 
method developed by Bruno (Bruno, 1985). The MCW-APM approach over- 
comes some of the problems found with present test design and scoring ap- 
proaches. Since resistance may be expected from major changes in testing^design 
and scoring from test administrators and test takers, it would be helpful to know 
the preferences of respondents who have been presented alternate format exams. 

To test the applicability of conjoint to this important area of concern in 
educational administration, alternate conjoint designs were developed and results 
presented for a large data set. The conjoint design was developed with four 
factors, and responses were received from 607 individuals. The testing illustra- 
tion displays conjoint's ability to incorporate analysis of age, grade level, test 
subject area, and ethnic background of the respondent in the interpretation of 
results. To further test the applicabiiity of the approach, the design was later 
modified to include five factors and aii analysis was performed on the number of 
additional questions required. 

Table 1 displays the four attributes and levels included in the initial analy- 
sis. These factors were identified from the literature on testing as important 
factors in test design (Bruno, 1989). Six pairwise designed questions were 
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presented to respondents for the four factors considered. The survey question- 
naire also solicited self-explicated respondent views for the four attributes. The 
coding and computation was similar to that identified in the Shukla study (1990). 

Responses were received from 607 students representing different levels of 
education and different colleges and universities. After elimination of responses 
which were incomplete and those which were identified by the researcher as 
incorrectly entered in data transfer from paper to disk, a total of 516 usable 
responses resulted. It is believed by the researcher that the majority of responses 
eliminated from the analysis represented errors in keyboard entry by prior indi- 
viduals rather than difficulty by respondents in responding to the survey ques- 
tions. As the complete survey consisted of 44 questions, the number of incom- 
plete responses are viewed as possibly representing difficulty respondents had 
with the length of the survey and not difficulty with conjoint's task complexity. 

Table 2 displays the aggregate results of attribute importance based upon 
the conjoint derived importance weights. The relative ranking of the attributes, 
by preference, was feedback first, partial credit second, restrictions third, and 
objectivity fourth. These relative rankings were consistent with the rankings of 
the attributes based upon self-explicated preferences identified. Tables 3 and 4 
identify the preference results based upon grade level and age, respectively. The 
two figures display a strong correlation, as one would expect, between prefer- 
ences derived from analysis of grade level and age. The interpretation of results 
suggests that as the grade level and age increase, the preference for feedback 
increases, and the preferences for no restrictions and objectivity decrease. T^ble 
5 displays results based upon subject area for the examination. The results indi- 
cate that feedback has a higher importance in quantitative subject areas such as 
mathematics, science, and economics and has lesser importance in verbal areas 
such as language and reading. Table 6 displays results based upon ethnic back- 
ground. From the results, it appears, that relative to other ethnic groups, by 
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attribute, the following conclusions could be presented: for feedback, Asian 
respondents had the highest importance weights and Hispanic respondents the 
least; for partial credit, Asian and White respondents had higher importance 
weights, and Hispanic and Black respondents with less importance weights; for 
no restrictions on choices, Asian respondents had higher importance weights and 
White respondents lesser importance weights; and for objectivity, Hispanic and 
Bliick respondents had higher importance weights and Asian respondents h id 
lesser importance weight. Based upon these results and examination of differ- 
ences among groups based upon age, grade, subject area, and ethnic background 
additional information is gained relative to just an aggregate analysis. 

To further assess the applicability of conjoint, the study design was modi- 
fied to consider the impact on the conjoint respondent task if a fifth factor was 
added. A fifth factor of test responding simplicity (need instructions on how to 
use answer sheet or no instructions required) was added to analyze the impact 
upon the number of questions required for completion by respondents. Original- 
ly with four factors, each at two level, six pairwise conjoint questions were 
required; with the fifth factor, also at two levels, a total of ten pairwise questions 
were required. This analysis illustrates conjoint's flexibility in permitting alter- 
nate designs. 

The testing design illustration displays that conjoint could be utilized for 
larger size samples and permits analysis by demographic factors. The illustration 
hf ^ps to show conjoint's applicability to education. 
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T^ble I 

Ti&sting Design Preference Attributes and Levels 



Attributes Levels 

Objectivity subjective 

objective 

Re.'itrictions restricted 

unrestricted 

Partial Credit allowed 

not allowed 

Feedback given 

not given 
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Aggregate Preferences for Testing Design 

Attribute % Importance 

Objectivity 22.9% 
Restrictions 23.3% 
Partial credit 26.0% 
Feedback 27.6% 
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Preferences for Ibsting Design by Grade 

Attribute 8-loth llth-12th CominColl University 

Objectivity 23.9% 23.7% 24.4% 21.9% 

Restrictions 23.0% 24.3% 23.2% 23.0% 

Partial Credit 26.5% 25.9% 24.8% 26.4% 

Feedback 26.4% 25.9% 27.4% 28.5% 



ERIC 



1^516 4 

Preferences for Tfesting Design by Age 

Attribute 
Objectivity 
Restrictions 
Partial Credit 
Feedback 



11-13 
24.0% 
22.2% 
24.9% 
28.6% 



14-18 
24.0% 
23.7% 
25.8% 
26.3% 



19-25 
21.8% 
23.4% 
26.5% 
28. 1% 



26+ 

22.4% 

22.5% 

25.7% 

29.2% 
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Table 5 

Preferences for Testing by Subject 

Attribute Math Science Language Economics Reading Ath}.. Battery 



Objectivity 


24. 


5% 


24.0% 


24.0% 


22.2% 


23. 


8% 


24.3% 


Restrictions 


21. 


9% 


19.7% 


24.5% 


25. S% 


24. 


4% 


23.5% 


Partial Credit 


26. 


3« 


27.1% 


27.1% 


23.3% 


27. 


2% 


25.5% 


Feedback 


27. 


0« 


29.0% 


24.2% 


28.8% 


24. 


4% 


26.4% 
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Preferences for Testing by Ethnicity 



Attribute 


White 


Black 


Hispanic 


Asian 


Objectivity 


22.6% 


23.5% 


23.8% 


21.5% 


Restrictions 


23.1% 


23.5% 


23.7% 


24.0% 


Partial Credit 


26.3% 


25.7% 


25.2% 


26.2% 


Feedback 


27.7% 


27.1% 


27.0% 


28.2% 
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